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Graduate schools must develop new criteria for identifying potentially talented 
individuals. Judgment analysis (JAN) uses a criterion of success, derived from the 
judgments of a group of experts, for graduate student selection. In considering a 
number of applicants for a position, a judge examines each applicants scores .or 
various profile variables or predictors relevant to a particular job. The judge :=tudie= 
the qualifications of each applicant and ranks each by means of an overall ccore or 
judgment. Each judgment thus becomes the applicant s criterion score for the set of 
profile Items or dimensions used as predictors. The procedure results m a senes of 
zero-order validity coefficients relating each individual predictor variable to the 
criterion. A computer analysis procedure reduces the number of judges, until all judges 
are grouped in a single cluster. Normative JAN and ipsative JAN. two variation^ of the 
JAN technique were studied. Normative JAN sought to determine the extent to which a 
policy regarding graduate admission standards existed among twenty representative 
faculty members. Each judge ranked the student on the basis of his profile variable^. 
Ipsative JAN used for its criterion variable the judges rankings or judgments based 
on personal knowledge as opposed to profile data information. (PS) 
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THE USE OF THE JUDGMENT ANALYSIS TECHNIQUE IN PREDICTING 

SUCCESS GRADUATE EDUCATION 

Samuel R. Houston and John T. Roscoe 

Today graduate schools are faced with the serious problem of finding 
and training more andh^fep graduate students for teaching and research. In 
view of this need, it would be naive to suggest that they raise their standards 
and select only the intellectual elite. By raising graduate admission standards 
and allowing only a select few to filter through graduate program, a substan- 
tial number of potentially talented individuals have been eliminated from gradu- 
ate study. Graduate schools instead must increase their resourcefulness and 
seek out new ways to identify potentially talented individuals who have escaped 
selection under the old criteria. 

A review of the literature reveals that there is a scarcity of studies 
related to the problem. For the most part, studies in predicting success in 
graduate education have followed the same basic design: aptitude and achieve- 
ment tests commonly required of applicants to graduate schools for the total 
population of a specified group are correlated with cumulative grade -point average 
(GPA), year-by-year GPA, GPA on various subsets of courses, and other varia- 



2 



tions of GPA. Coefficients of correlation have been statistically significant but 
they have not been efficient in forecasting power. In general, studies have 
grouped graduate students together and have ignored the variable of the major 
department in which graduate students are enrolled. Criticisms of using the 
GPA as criterion of success have been made both in terms of its appropriate- 
ness as well as with the limited range of grades in graduate school. 

JUDGMENT ANALYSIS 

A new promising technique which avoids the criticisms mentioned 
above was described in 1961. Now called Judgment Analysis or JAN, it uses 
a criterion of success derived from the judgments of a group of experts. JAN 
is an adaptation of methods developed by Ward and by Bottenberg and Christal 
which groups criteria in terms of the homogeneity of their prediction equations. 
It gives the admitting school a standard for deciding whether to accept a given 
student: that standard is a policy which they have created themselves. The 
policy is described by the weights to be applied to scores obtained on the various 

profile items or admissions data. 

How is the JAN technique used? In considering a number of applicants 
for a position, a judge examines each applicant's scores for various profile 
variables or predictors relevant to a particular job. Instead of evaluating the 
relative importance of each profile item and weighting the predictor variables to 
form a selection battery, the judge studies the qualifications of each applicant 
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judgment. Each judgment thus be- 



and ranks each by means of an overall score or 
comes the applicant’s criterion score for the set of profile items or dimensions 
used as predictors. The procedure results in a series of zero-order validity co- 
efficients relating each individual predictor variable to the criterion. 

After the criterion scores have been recorded for all the judges, the first 
step in the analysis procedure is to compute a least squares solution of a multiple 
regression equation to predict the criterion scores given by each judge, usmg the 
profile items as predictors. Christal (1963) discusses the criterion -grouping tech- 
niques in some detail. Using the computed for each individual, unacceptable 
raters may be eliminated by comparing the R^s computed from their equations with 
the r2s obtained for the other judges in the sample. Next, a single value of r 2 is 
computed to indicate the overall predictive efficiency when all the individual judges 
who have the most homogeneous equations are located. The computer prints the 
single equation that best represents the joint policy of these two judges as well as 
the loss in overall predictive efficiency that results when the N original equations 
are reduced to N-1 equations. The process continues systematically to reduce the 
number of judges by one at each step until all judges have been grouped into a single 
cluster. At each step, examination of the loss in overall predictive efficiency (the 
reduction in r 2) makes it possible to Identify the different policies which may exist. 

■NORMATIVE JAN 

Tv/o variations of the JAN technique for predicting success in graduate 
school were investigated-Normative JAN and Ipsative JAN. Normative JAN sought 
to determine the extent to which a polity regarding graduate admission standards 



existed among twenty representative graduate faculty members at Colorado State 
College. The selected graduate faculty members who served as judges responded 
to profile data on thirty randomly selected doctoral graduates who graduated between 
1963-1966. Each judge was asked to rank each student on the basis of his profile 
variables with one of five scores . A score of fi^ indicated that the judge thought 
the student would probably graduate in the top fifth of the class for a particular year; 
a score of one indicated a judgment that the student would graduate in the lowest fifth. 
Sixteen of the judges, that is, 80 percent responded. 

LIST OF NORMATIVE JAN VARIABLES 

1) Normative -JAN criterion 

2) Colorado State College master *s degree 

3) Cumulative grade point average 

4) Years between B.A. and M.A. degree 

5) Rating on Interview 

6) Recommendation on Interview 

7) Sex 

8) GRE -Verbal score 

9) GRE -Quantitative score 

10) GRE -Social Science score 

11) GRE -Humanities score 

12) GRE -Natural Science score 

13) GRE -Advanced Education score 

14) English Usage Test score 

15) Education Major 

16) Science Major 

17) Psychology Major 

18) Business Major 

19) Music Major 

20) Industrial Arts Major 

21) Physical Education Major 

22) Other Major Field 

Basically, the profile variables can be grouped into three subsets: 

(a) biographical data (variables 2-7 inclusive); (b) test data (variables 8-14 inclusive) 
and (c) major field data (variables 15-22 inclusive). The predictor variables were 
selected because they represented readily accessible information which might have 



some predictive relationship to a criterion measure of graduate school success; in 
this case, die criterion measure utilized was the judgment score expressed by die 
graduate faculty members serving as judges . The diree highest zero-order correla- 
tions between predictor variables and die criterion variable are Other Major Field 
(-.83), Physical Education Major (-.71) and Music Major (-.66). 

A least squares solution of a multiple regression equation to predict die 
criterion decisions given by each judge gave an R of .8580 which indicates the overall 
predictive efficiency. A multiple correlation using die twenty-one independent variables 
as predictors of the Normative-JAN criterion was .93. Not only is this significant at 

the .01 level, but it is exceptionally high from a predictive viewpoint: approximately 
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86 percent of the criterion variance is thereby explained. The high R indicates that 

the interrater agreement is exceptionally high — especially considering the categorical 

criterion - - and that, essentially, one policy is being expressed by the judges . 

In order to investigate the efficiency of the prediction when certain groups 

of predictors were removed from the system, the investigators used multiple linear 

regression, according to Ward (1962), to determine the unique contribution of proper 

subsets of the predictor variables, (2-22), to the prediction of die criterion. The 

contribution of a set of variables to prediction may be measured by the difference 
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between two squares of multiple correlation coefficients (R s), one obtained for a full 

regression model (FM), and die other obtained for a regression equation in which 

the proper subset of variables under consideration has been deleted; this model is 
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called the restricted model, (RM). The difference between the two R s may be 
tested for statistical significance with the variance ratio test . 

Approximately 59 percent (.5928) of the criterion variance is estimated 



to be attributable to the tliirteen variables which essentially consist of biographical 
and test data and this in turn, leads to an estimate of approximately .26 for the 
unique contribution of variables (15-22) as a group. This was significant at the .01 
level . Similarly, the unique contribution of the seven predictor variables involving 
test data (8-14) is estimated at .0453 and was significant at the .01 level. Finally, 
an estimate of .0377 is given as the unique contribution of the six predictor variables 
involving biographical data (2-7) and this contribution was significant at the .01 level. 
IPSATIVE JAN 

Ipsative JAN used for its criterion variable the rankings or judgments sub- 
mitted by the judges who ranked the doctoral graduates on the basis of personal 
knowledge as opposed to profile data information. The judges were requested to 
select the names of ten students about whom they were knowledgeable and then to 
rank (without access to profile data information) these students in terms of their 
estimate of the student*s professional promise. The names of the students were 
selected from a list of 231 recent doctoral graduates at Colorado State College during 
the same period, 1963-1966. Sixteen judges, or 80 percent, responded; however, one 
judge ranked only eight students . 

LIST OF IPSATIVE JAN VARIABLES 

1) Ipsative -JAN criterion 

2) Colorado State College master *s degree 

3) Cumulative grade point average 

4) Years between B.A. and M.A. degree 

5) Age doctorate received 

6) Sex 

7) GRE -Verbal score 

8) GRE -Quantitative score 

9) GRE -Social Science score 

10) GRE -Humanities score 

11) GRE -Natural Science score 

12) GRE -Advanced Education score 
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13) English Usage Test score 

14) Education Major 

15) Science Major 

16) Psychology Major 

17) Business Major 

18) Music Major 

19) Industrial Arts Major 

20) Physical Education Major 

21) Other Major Field 

Essentially, the three groups of predictor variables investigated were (a) 
biographical data (variables 2-6 inclusive); (b) test data (variables 7-13 inclusive); 

(c) major field data (variables 14-21 inclusive). The biographical items in the 
Ipsative JAN study differ slightly from the biographical variables in the Normative JAN 

study because of missing data . 

The Ipsative JAN technique was not used to determine the extent to which a 
policy may or may not be present. It is doubtful that one could claim a policy exists 
in terms of the predictor variables when each judge was requested to give his rating 
on the basis of a subjective knowledge of the individual and not on the basis of any 
profile data (predictors). Primarily, the purpose was to determine the amount of the 
relationship between the available predictor information and the Ipsative JAN criterion. 

The three highest zero -order correlations between the predictor variables 
and the Ipsative JAN criterion variable are GRE-Advanced Education (.30), GRE- 
Verbal (.26), and English Usage (.24). The 20 -variable composite accounts for 
.1700 of the total Ipsative JAN criterion variance . A multiple correlation using the 
twenty independent variables as predictors of the criterion was .42. While this is 
significant at the .01 level, it is low from a predictive point of view; approximately 
83 percent of the variance is unaccounted for when the twenty variables are used as 



predictors . 
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Approximately 16 percent (.1591) of the criterion variance is estimated 
to be attributable to the thirteen variables which essentially consist of biograpliical 
(2-6) and test data (7-13). Tlie loss in predictive accuracy when major subject field 
data are not considered does not seem sufficient to be of practical importance. Tlie 
unique contribution of die eight predictor variables (14-21) involved with identifying 
the major subject field was not significant at die .01 level. A similar approach shows 
that the unique contribution of die seven predictor variables involving test data (7-13) 
is estimated at .0700. The loss in predictive accuracy when test data are not con- 
sidered seems sufficient to be of practical importance even though die unique contri- 
bution of the seven predictor variables involving test data information was not 
significiant at the .01 level. An estimate of .0104 is given as the unique contribution 
of the five predictor variables involving biographical data (2-6). This contribution was 
not significant at the .01 level; the loss in predictive accuracy when biographical data 
are not considered does not seem sufficient to be of practical importance. 

SUMMARY 

Normative JAN develops a faculty policy for graduate admission and deter- 
mines the minimum numJier of policies that may be expressed. The investigators were 
more interested in determining t o what extent a policy had been expressed than in 
actual regression equations. Consequently, no attempt was made to give an exhaustive 
analysis of the set and various subsets of predictor variables . Instead, the predictor 
variables were grouped into three proper disjoint subsets and the unique contribution 
of each of these subsets was investigated. Results seem to indicate that, essentially, 
one policy was expressed by the judges using the Normative JAN technique . Further- 
more, each of the diree disjoint proper subsets contributed significantly to the 



9 



prediction of the Normative JAN criterion. This suggests diat this kind of admission 
profile data provides relevant information for judges in policy development. 

The Ipsative JAN variation generates an additional criterion of graduate 
school success; namely, a graduate faculty member*s estimate of a student*s pro- 
fessional promise. Results were statistically significant though weak from a 
predictive viewpoint. The multiple correlation obtained, with judgments of graduate 
faculty members used as a criterion, is essentially the same as correlations obtamed 
when cumulative graduate grade point averages are utUized. Each of the three disjoint 
proper subsets failed to contribute significantly to the prediction of the Ipsative JAN 
criterion even though in the case of test data the loss 'in predictive accuracy seems 
sufficient to be of practical importance . It might well be that the results of the 
Ipsative JAN study could provide additional relevant information for judges developing 
a policy for graduate admission via Normative JAN. 
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